1. Field of the Invention
The present invention generally relates to robotics and medical imaging techniques and, more particularly to robotically-assisted surgical systems and other devices incorporating methods for registering image data (both pre-operative and intra-operative) to physical space and for providing feedback, and in particular visual feedback, to the clinician.
2. Description of the Related Art
Computers are increasingly used to plan complex surgeries by analyzing pre-operative Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) scans of a patient.
To execute the surgical plan, it is important to accurately align or register the three-dimensional pre-operative and intra-operative data to an actual location of the patient""s anatomical features of interest during surgery.
One conventional technique for performing this type of registration is to attach a stereo-tactic frame or fiducial markers to the patient, and to precisely locate the frame or markers prior to and during surgery.
For example, in the case of a surgery involving a patient""s femur, a conventional registration protocol includes implanting three metallic markers or pins in the patient""s femur (e.g., one proximally in the trochanter and two distally in the condyles, near the knee). However, the insertion of the pins requires minor surgery. A CT-scan image of the patient is subsequently acquired. By analyzing the CT data, the surgeon decides upon the size and location of the implant that best fits the patient""s anatomy. During surgery, the metallic pins are exposed at the hip and knee. The patient""s leg is attached to a surgical robot device that then must locate the exposed pins. A registration, or coordinate transformation from CT space to robot space, is computed using the locations of the three pins as a Cartesian frame. The accuracy of this registration has been measured to be better than one millimeter. This conventional registration protocol is described in U.S. Pat. No. 5,299,288 entitled xe2x80x9cIMAGE-DIRECTED ROBOTIC SYSTEM FOR PRECISE ROBOTIC SURGERY INCLUDING REDUNDANT CONSISTENCY CHECKINGxe2x80x9d by Glassman et al., and incorporated herein by reference.
However, using such pins as markers is not always desirable, as they may cause significant patient discomfort, and the required surgical procedure to insert and subsequently remove the pins is inconvenient and costly to the patient.
An alternative registration technique is to perform anatomy-based registration that uses anatomical features of the patient (e.g., generally bone features), as markers for registration. Conventional methods for performing anatomy-based registration are described in xe2x80x9cRegistration of Head CT Images to Physical Space Using a Weighted Combination of Points and Surfacesxe2x80x9d by Herring et al., in IEEE Transactions on Medical Imaging, Vol. 17,No. 5, pages 753-761, 1998 and in U.S. patent application Ser. No. 08/936,935 entitled xe2x80x9cMETHODS AND APPARATUS FOR REGISTERING CT-SCAN DATA TO MULTIPLE FLUOROSCOPIC IMAGESxe2x80x9d, filed on Sep. 27, 1997 by A Gueziec et al., each of which is herein incorporated by reference in its entirety.
Once the registration has been performed, it is important to provide the clinician with means to assess the registration, allowing him or her to validate, reject or improve the registration (and the surgical plan). A system and method for advising a surgeon is described in U.S. Pat. No. 5,445,166, entitled xe2x80x9cSYSTEM FOR ADVISING A SURGEONxe2x80x9d, by Taylor, which is herein incorporated by reference in its entirety. Taylor describes a system for guiding the motions of a robot, or of a positioning device controlled by motors, and teaches how audio feedback and force feedback can be provided to a surgeon. Taylor also describes a visual adviser allowing comparison of the surgical plan with its execution. The system taught by Taylor optionally uses a camera at the end of a surgical instrument that sends an image to the graphics adapter, optionally mixed with graphics output of the computer.
A conventional technique for simulating a post-operative X-ray image is described in xe2x80x9cAn Overview of Computer-Integrated Surgery at the IBM T. J. Watson Research Centerxe2x80x9d by Taylor et al., in IBM Journal of Research, 1996, which is herein incorporated by reference in its entirety.
Thus, conventional techniques are useful for registering three-dimensional pre-operative and intra-operative data to an actual location of anatomical features of interest during surgery, and to provide advice to the surgeon. However, none of the conventional techniques teaches how to simulate a post-operative condition depending upon the registration of image data to physical space, by fusing intra-operative images with registered pre-operative data, and generating new images.
In Taylor et al., the simulated post-operative X-ray image is generated using only pre-operative CT (Computed Tomography) data. Herring et al. do not teach how to evaluate the registration accuracy intra-operatively.
Although Glassman et al.""s and Taylor""s systems compare a surgical plan and its execution, neither Glassman et al. nor Taylor teaches how to simulate the outcome of a surgical plan prior to the actual execution of the plan. With Taylor""s system, a surgeon can take corrective measures to minimize the effects of a wrongful execution of the plan, but cannot make a decision before any execution of the plan and therefore cannot prevent all errors before they occur.
Further, the information produced by Taylor""s system for advising a surgeon is not represented in the form of conventional medical media (e.g., such as X-ray images) and requires an extra burden on the surgeon in order to interpret and evaluate this information.
Thus, it is believed that conventional techniques do not exist (or at the very least are inadequate) for (a) providing the surgeon with post-operative evaluations prior to surgery, that are obtained by merging intra-operative image data and pre-operative data, and (b) presenting such evaluations in a standard clinical fashion (e.g., such as augmented X-ray images) that is natural for a surgeon to interpret.
Other problems of the conventional systems and methods include the limited availability of 2-D/3-D registration methods in conventional art systems for advising a surgeon.
In another conventional system, as described in the above-mentioned U.S. patent application Ser. No. 09/299,643, the geometric distortion of an X-ray image is always corrected. This is problematic because a clinician or surgeon is used to seeing the unmodified image (e.g., an image with distortion). That is, as a practical matter, surgeons generally are not familiar with seeing the modified image. Surgeons are used to interpreting the unmodified images. Further, such a correction may cause image degradation or blurring due to the reformatting of the image. Additionally, slower and more complex processing results from the image correction process.
In view of the foregoing and other problems of the conventional methods and structures, an object of the present invention is to provide a method and structure for intra-operatively providing the surgeon with visual evaluations of possible surgical outcomes ahead of time, the evaluations being obtained by merging intra-operative image data and pre-operative data, and presented in a standard clinical fashion (e.g., such as augmented X-ray images) that is natural and easy for a surgeon to interpret.
Another object of the present invention is to provide a system and method for fusing three-dimensional shape data on distorted images without correcting for distortion.
Yet another object of the present invention is to provide a system and method for assisting the surgeon in improving an inaccurate registration of a pre-operative surgical plan to a physical space of an operating room.
Still another object of the present invention is to provide an improved robotically assisted surgical system that also provides visual post-operative evaluations.
A further object of the present invention is to provide an improved robotically-assisted surgical system that includes a system for assisting the surgeon in improving a registration.
Another object of this invention is to provide an improved robotically assisted surgical system that includes a system for preventing surgical errors caused by internal failure of the robot""s calibration system.
In a first aspect of the present invention, a system is provided for fusing three-dimensional shape data on distorted images without correcting for distortion.
The inventive system preferably includes a data processor. The data processor takes as inputs a three-dimensional surgical plan or three-dimensional plan of therapy delivery, one or a plurality of two-dimensional intra-operative images, a three-dimensional model of pre-operative data, registration data, and image calibration data.
The data processor produces one or a plurality of simulated post-operative images, by integrating a projection of a three-dimensional model of pre-operative data onto one or a plurality of two-dimensional intra-operative images, without correcting for any distortion in the images.
The data processor optionally receives an input from a surgeon or a clinician. The input preferably includes a set of constraints on the surgical plan or plan of therapy delivery. The data processor preferably optimizes the surgical plan or plan of therapy delivery using the constraints.
In another aspect of the present invention, a system (and method) for generating simulated data, includes a medical imaging camera for generating images, a registration device for registering data to a physical space, and to the medical imaging camera, and a fusion (integration) mechanism for fusing (integrating) the data and the images, without correcting for distortion to generate simulated data.
In yet another aspect of the invention, a method of fusing three-dimensional image data on distorted images, includes receiving a potentially distorted image, calibrating the potentially distorted image, based on the calibration, computing an apparent contour of the three-dimensional shape of the potentially distorted image, for each pixel of the image, determining a ray in 3-dimensional space and computing a distance from the ray to the apparent contour, and selectively adjusting a pixel value of the potentially distorted image based on the distance.
In yet another aspect of the invention, a signal-bearing medium is provided for storing a program for performing the method of the invention. Other aspects of the invention are also set forth below.
With the invention, the surgeon is provided with intra-operative visual evaluations of possible surgical outcomes in advance, with the evaluations being obtained by merging intra-operative image data and pre-operative data. Such evaluations are presented in a standard clinical fashion that is natural and easy for a surgeon to interpret. Further, the inventive system compares several registration methods of pre-operative data to the physical space of the operating room.
Moreover, the invention assists the surgeon in improving an inaccurate registration of a pre-operative surgical plan to the physical space. Additionally, the system can be robotically-assisted and can provide visual post-operative evaluations.
Additionally, in the robotically-assisted implementation of the inventive system, surgical errors, caused by internal failure of the robot""s calibration system, can be prevented.
Further, with the invention, a clinician or surgeon can view images in the manner that they are accustomed. That is, the clinician or surgeon can view the unmodified image (e.g., an image with distortion), in the manner with which they are familiar. Thus, the surgeons can continue to interpret the unmodified images, as is customary. Further, since no correction is performed as in the conventional system and methods, no image degradation or blurring results from such image correction and reformatting of the image. Additionally, processing speed is not decreased, and similarly processing resources are not increased since the processing of the method of the present invention is less complex than that of the conventional systems and methods.